Measuring patient mobility in the ICU using a novel non-invasive sensor

ABSTRACT

An embodiment in accordance with the present invention includes a technology to continuously measure patient mobility automatically, using sensors that capture color and depth images along with algorithms that process the data and analyze the activities of the patients and providers to assess the highest level of mobility of the patient. An algorithm according to the present invention employs the following five steps: 1) analyze individual images to locate the regions containing every person in the scene (Person Localization), 2) for each person region, assign an identity to distinguish ‘patient’ vs. ‘not patient’ (Patient Identification), 3) determine the pose of the patient, with the help of contextual information (Patient Pose Classification and Context Detection), 4) measure the degree of motion of the patient (Motion Analysis), and 5) infer the highest mobility level of the patient using the combination of pose and motion characteristics (Mobility Classification).

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a 35 U.S.C. § 371 U.S. national entry ofInternational Application PCT/US2017/055108, having an internationalfiling date of Oct. 4, 2017, which claims the benefit of U.S.Provisional Application No. 62/403,890, filed Oct. 4, 2016, the contentof each of the aforementioned applications is herein incorporated byreference in their entirety.

FIELD OF THE INVENTION

The present invention relates generally to image-based medicalinformatics. More particularly, the present invention relates tomeasuring patient mobility in the ICU using a novel non-invasive sensor.

BACKGROUND OF THE INVENTION

Monitoring human activities in complex environments are finding anincrease in interest. In 2012, the Institute of Medicine released theirlandmark report on developing digital infrastructures that enable rapidlearning health systems; one of their key postulates is the need forimprovement technologies for measuring the care environment. Currently,simple measures such as whether the patient has moved in the last 24hours, or whether the patient has gone unattended for several hoursrequire manual observation by a nurse, which is highly impractical toscale. Early mobilization of critically ill patients has been shown toreduce physical impairments and decrease length of stay, however thereliance on direct observation limits the amount of data that may becollected. Accurate measurement of patient mobility, as part of routinecare, assists in evaluating early mobilization and rehabilitation andhelps to understand patient exposure to the harmful effects of bedrest.

To automate this process, non-invasive low-cost camera systems havebegun to show promise, though current approaches are limited due to theunique challenges common to complex environments. First, though persondetection in images is an active research area, significant occlusionspresent limitations, because the expected appearances of people do notmatch what is observed in the scene. Part-based deformable methods dosomewhat address these issues as well as provide support forarticulation, however when combining deformation with occlusion, thesetoo suffer for similar reasons.

In research, patient mobility measurement may be performed via directobservation by a trained and dedicated researcher. Direct observationtechniques, such as behavioral mapping, provide comprehensivedescriptive data sets and are more accurate than retrospective report,but are labor-intensive, thus limiting the amount and duration of datacollection. If evaluated as part of routine clinical care, mobilitystatus is typically estimated using a mobility scale and recorded onceor twice daily. However, such discrete subjective recordings of apatient's maximal level of mobility over a 12 or 24 hour time period aresubject to recall bias and not truly representative of a patient'soverall mobility level (e.g., a patient may achieve a maximal mobilitylevel, such as standing, only for a couple of minutes in a day). Thus,accurate manual measurement and recording of mobility level is notfeasible for whole-day observation.

Currently, few techniques exist to automatically and accurately monitorICU patient's mobility. Accelerometry is one method that has beenvalidated, but it has limited use in critically ill inpatientpopulations. Related to multi-person tracking, methods have beenintroduced to leverage temporal cues, however hand-annotated regions aretypically required at the onset, limiting automation. To avoid manualinitializations, techniques such as employ a single per-frame detectorwith temporal constraints. Because single detectors are limited towardsappearance variations, proposes to make use of multiple detectors,however this assumes that the spatial configuration between thedetectors is fixed, which does not scale to address significant posevariations.

Much activity analysis research has approached action classificationwith bag-of-words approaches. Typically, spatio-temporal features, suchas Dense Trajectories, are used with a histogram of dictionary elementsor a Fisher Vector encoding. Recent work has applied ConvolutionalNeural Network (CNN) models to the video domain by utilizing bothspatial and temporal information within the network topology. Other workuses Recurrent Neural Networks with Long Short Term Memory to modelsequences over time.

Accordingly, there is a need in the art for a non-invasive automatedapproach to measuring patient mobility and care processes due to theadvent of inexpensive sensing hardware and low-cost data storage, andthe maturation of machine learning and computer vision algorithms foranalysis.

SUMMARY OF THE INVENTION

The foregoing needs are met, to a great extent, by the present inventionwhich provides a method of measuring patient mobility within a roomincluding analyzing individual images from a RGB-D sensor to locate aregion containing each person in the room. The method includes assigningan identity for each person in the region, to distinguish ‘patient’ vs.‘not patient’. The method also includes determining a pose of thepatient, with contextual information_and measuring a degree of motion ofthe patient. Additionally, the method includes inferring the highestmobility level of the patient using the combination of pose and motioncharacteristics.

In accordance with an aspect of the present invention, the methodincludes generating a volumetric representation of patient movementaround the room. The method can also include generating a heatmap ofpatient movement throughout the room and generating a heatmap ofmovement of regions of the patient's body. The method includesclassifying the pose of the patient into 4 discrete categories: (1)lying in bed, (2) sitting in bed, (3) sitting in chair, and (4)standing. The method includes classifying patient motion as “in-bedactivity” if the patient's total body speed signature exceeds athreshold. Additionally, the method includes classifying patient motionas “nothing in bed” if the patient's total body speed signature is belowa threshold. The method includes executing the method with anon-transitory computer readable medium. The method also includes apredetermined area for measuring patient mobility and defining thepredetermined area for measuring patient mobility as the patient room.

In accordance with another aspect of the present invention, a system formeasuring patient mobility within a room includes an RGB-D sensor. Thesystem also includes a non-transitory computer readable mediumprogrammed for analyzing individual images from the RGB-D sensor tolocate a region containing each person in the room. The non-transitorycomputer readable medium is programmed for assigning an identity foreach person in the region, to distinguish ‘patient’ vs. ‘not patient’and determining a pose of the patient, with contextual information.Additionally, the non-transitory computer readable medium is programmedfor measuring a degree of motion of the patient and inferring thehighest mobility level of the patient using the combination of pose andmotion characteristics.

In accordance with still another aspect of the present invention, thesystem includes generating a volumetric representation of patientmovement. The system includes generating a heatmap of patient movementand generating a heatmap of movement of regions of the patient's body.The system includes classifying the pose of the patient into 4 discretecategories: (1) lying in bed, (2) sitting in bed, (3) sitting in chair,and (4) standing. Additionally, the system includes classifying patientmotion as “in-bed activity” if the patient's total body speed signatureexceeds a threshold. The system includes classifying patient motion as“nothing in bed” if the patient's total body speed signature is below athreshold. The system includes assigning a numerical mobility value tothe patient mobility. The system includes defining a predetermined areafor measuring patient mobility and defining the predetermined area formeasuring patient mobility as the patient room.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings provide visual representations, which will beused to more fully describe the representative embodiments disclosedherein and can be used by those skilled in the art to better understandthem and their inherent advantages. In these drawings, like referencenumerals identify corresponding elements and:

FIG. 1 illustrates a flowchart of mobility prediction framework,according to an embodiment of the present invention.

FIGS. 2A-2D illustrate image views from full-body and Head detectors.

FIG. 3 illustrates a three-dimensional view of a patient room with aheatmap of movement within the room, according to an embodiment of thepresent invention. In another representation of the data, an overheadview is provided to show movement throughout the patient room.

FIG. 4 illustrates a top down view of a patient room with a heatmap ofmovement within the room, according to an embodiment of the presentinvention.

FIG. 5 illustrates a sensor system, according to an embodiment of thepresent invention in an ICU room and example color (converted tograyscale for demonstration) and depth images captured by the sensors.

FIG. 6A illustrates graphical views of patient mobility and image viewsfrom a patient room according to an embodiment of the present invention.

FIG. 6B illustrates a flow diagram view of an algorithm according to anembodiment of the present invention.

FIG. 7 illustrates graphical and image views of how the presentinvention can incorrectly measure a patient's mobility when thecaregiver is close to the patient.

FIG. 8 illustrates a graphical view of how the present inventioncontinuously measures patient mobility.

DETAILED DESCRIPTION

The presently disclosed subject matter now will be described more fullyhereinafter with reference to the accompanying Drawings, in which some,but not all embodiments of the inventions are shown. Like numbers referto like elements throughout. The presently disclosed subject matter maybe embodied in many different forms and should not be construed aslimited to the embodiments set forth herein; rather, these embodimentsare provided so that this disclosure will satisfy applicable legalrequirements. Indeed, many modifications and other embodiments of thepresently disclosed subject matter set forth herein will come to mind toone skilled in the art to which the presently disclosed subject matterpertains having the benefit of the teachings presented in the foregoingdescriptions and the associated Drawings. Therefore, it is to beunderstood that the presently disclosed subject matter is not to belimited to the specific embodiments disclosed and that modifications andother embodiments are intended to be included within the scope of theappended claims.

An embodiment in accordance with the present invention includes a systemto continuously measure patient mobility automatically, using sensorsthat capture color and depth image data. The system also processes thecolor and depth image data to analyze the activities of the patients andproviders to assess the highest level of mobility of the patient.Analysis according to the present invention employs the following fivesteps: 1) analyze individual images to locate the regions containingevery person in the scene (Person Localization), 2) for each personregion, assign an identity to distinguish ‘patient’ vs. ‘not patient’(Patient Identification), 3) determine the pose of the patient, with thehelp of contextual information (Patient Pose Classification and ContextDetection), 4) measure the degree of motion of the patient (MotionAnalysis), and 5) infer the highest mobility level of the patient usingthe combination of pose and motion characteristics (MobilityClassification).

FIG. 1 shows an overview of a system of the present invention. Peopleare localized, tracked, and identified within a predetermined spaceusing an RGB-D sensor. The predetermined space can be a room, a regionof a room, or other space where tracking may be useful. The parameterscan be walls of the room or dimensions entered into the system to definethe tracking area. FIG. 1 illustrates a flowchart of mobility predictionframework, according to an embodiment of the present invention. Thesystem tracks people in the patient's room, identifies the “role” ofeach (“patient”, “caregiver”, or “family member”), relevant objects, andbuilds attribute features for mobility classification. The presentinvention predicts the pose of the patient and identifies nearby objectswithin the predetermined space to serve as context. Finally, in-placemotion is analyzed and a classifier trained to determine the highestlevel of patient mobility.

The tracking method works by formulating an energy function havingspatial and temporal consistency over multiple part-based detectors (seeFIGS. 2A-2D). FIGS. 2A-2D illustrate image views from full-body and Headdetectors. The head detector may fail with proximity or distance, asillustrated in FIGS. 2A and 2D, respectively. The full-body detector mayalso struggle with proximity, as illustrated in FIGS. 2B and 2C,respectively. (To protect privacy, all images are blurred).

The relationship between detectors within a single frame is modeledusing a deformable spatial model and then tracked in an online setting.Modeling Deformable Spatial Configurations: For objects that exhibitdeformation, such as humans, there is an expected spatial structurebetween regions of interest (ROIs) (e.g., head, hands, etc.) across posevariations. Within each pose (e.g. lying, sitting, or standing), an ROI(e.g. head) can be estimated based on other ROIs (e.g. full-body). Tomodel such relationships, there is a projection matrix A_(ll) ^(c),which maps the location of ROI l to that of l₀ for a given pose c. Witha training dataset, C types of poses are determined automatically byclustering location features, and projection matrix A_(ll) ^(c), can belearnt by solving a regularized least-square optimization problem.

To derive the energy function of the deformable model, the number ofpersons in the t-th frame is denoted as M^(t). For the m-th person, theset of corresponding bounding boxes from L ROIs is defined by X^(t)={X₁^(t)(m), . . . , X_(L) ^(t)(m)}. For any two proposed bounding boxesX_(l′) ^(t)(m) and X_(l) ^(t)(m) at frame t for individual m, thedeviation from the expected spatial configuration is quantified as theerror between the expected location of the bounding box for the secondROI conditioned on the first. The total cost is computed by summing, foreach of the M^(t) individuals, the minimum cost for each of the Csubcategories:

$\begin{matrix}{{E_{spa}\left( {X^{t},M^{t}} \right)} = {\sum\limits_{m = 1}^{M^{t}}\;{\min\limits_{1 \leq c \leq C}{\Sigma_{l \neq l^{\prime}}{{{A_{{ll}^{\prime}}^{c}{X_{l}^{t}(m)}} - {X_{l^{\prime}}^{t}(m)}}}^{2}}}}} & (1)\end{matrix}$

Grouping Multiple Detectors:

The process of detecting people to track is automated using acombination of multiple part-based detectors. A collection of existingdetection methods can be employed to train K detectors; each detector isgeared towards detecting an ROI. Let us consider two bounding boxesD_(k) ^(t)(n) and D_(k′) ^(t)(n′) from any two detectors k and k′,respectively. If these are from the same person, the overlapped regionis large when they are projected to the same ROI using a projectionmatrix. In this case, the average depths in these two bounding boxes aresimilar. The probability that these are from the same person iscalculated as:p=ap _(over)+(1−a)p _(depth)  (2)where a is a positive weight, p_(over) and p_(depth) measure theoverlapping ratio and depth similarity between two bounding boxes,respectively. These scores are

$\begin{matrix}{p_{over} = \left( {\frac{{{A_{{l{(k)}}{l{(k^{\prime})}}}^{c}{D_{k}^{t}(n)}}\bigcap{D_{k^{\prime}}^{t}\left( n^{\prime} \right)}}}{\min\left( {{{A_{{l{(k)}}{l{(k^{\prime})}}}^{c}{D_{k}^{t}(n)}}},{{D_{k^{\prime}}^{t}\left( n^{\prime} \right)}}} \right)},\frac{{{D_{k}^{t}(n)}\bigcap{A_{{l{(k^{\prime})}}{l{(k)}}}^{c}{D_{k^{\prime}}^{t}\left( n^{\prime} \right)}}}}{\min\left( {{{D_{k}^{t}(n)}},{{A_{{l{(k^{\prime})}}{l{(k)}}}^{c}{D_{k^{\prime}}^{t}\left( n^{\prime} \right)}}}} \right)}} \right)} & (3) \\{\mspace{76mu}{p_{depth} = {{\frac{1}{2}e^{\frac{- {({{v_{k}^{t}{(n)}} - {v_{k^{\prime}}^{t}{(n^{\prime})}}})}^{2}}{2{\sigma_{k}^{t}{(n)}}^{2}}}} + {\frac{1}{2}e^{\frac{- {({{v_{k}^{t}{(n)}} - {v_{k^{\prime}}^{t}{(n^{\prime})}}})}^{2}}{2{v_{k^{\prime}}^{t}{(n^{\prime})}}^{2}}}}}}} & (4)\end{matrix}$where l maps the k-th detector to the l-th region-of-interest, v and 6denote the mean and standard deviation of the depth inside a boundingbox, respectively.

By the proximity measure given by (2), the detection outputs are groupedinto N^(t) sets of bounding boxes. In each group G^(t)(n), the boundingboxes are likely from the same person. Then, a cost function thatrepresents the matching relationships between the true positions of thetracker and the candidate locations suggested by the individualdetectors are defined as:

$\begin{matrix}{{E_{\det}\left( {X^{t},M^{t}} \right)} = {\sum\limits_{n = 1}^{N^{t}}\;{\min\limits_{1 \leq m \leq M^{t}}{\sum\limits_{{D_{k}^{t}{(n^{\prime})}} \in {G^{t}{(n)}}}{{w_{k}^{t}(n)}{{{D_{k}^{t}\left( n^{\prime} \right)} - {X_{l{(k)}}^{t}(m)}}}^{2}}}}}} & (5)\end{matrix}$where w_(k) ^(t)(n) is the detection score as a penalty for eachdetected bounding box. Tracking Framework: The tracker is initialized attime t=1 by aggregating the spatial (Eq. 1) and detection matching (Eq.5) cost functions. To determine the best bounding box locations at timet conditioned on the inferred bounding box locations at time t−1, thetemporal trajectory E_(dyn) and appearance E_(app) energy functions areextended and the joint optimization solved as:

$\begin{matrix}{{\min\limits_{X^{t},M^{t}}{\lambda_{\det}E_{\det}}} + {\lambda_{spa}E_{spa}} + {\lambda_{exc}E_{exc}} + {\lambda_{reg}E_{reg}} + {\lambda_{dyn}E_{dyn}} + {\lambda_{app}E_{app}}} & (6)\end{matrix}$

TABLE 1 Sensor Sensor Scale ICU Mobility Scale^([10]) Label A. Nothingin bed 0. Nothing, lying in bed (i) i B. In-bed activity 1. Sitting inbed (ii), exercises in bed (iii) ii iii C. Out-of-bed 2. Passively movedto chair (no standing) iii→v/ vii→iv activity 3. Sitting over edge ofbed (iv) iv 4. Standing (v) v 5. Transferring bed to chair (withstanding) iii→v/ vii→iv 6. Marching (vi) in place (at bedside) for vishort duration D. Walking (vii) 7. Walking with assistance of 2 or morevii people 8. Walking with assistance of 1 person 9. Walkingindependently with a gait aid 10. Walking independently without a gaitaidTable 1.

Table comparing the Sensor Scale, containing the 4 discrete levels ofmobility that the present invention is trained to categorize from avideo clip of a patient in the ICU, to the standardized ICU MobilityScale, used by clinicians in practice today.

Patient Identification:

A pre-trained CNN is fine-tuned, which is initially trained on ImageNet(http://image-net.org/). From the RGB-D sensor, the color images areused to classify images of people into one of the following categories:patient, caregiver, or family-member. Given each track from themulti-person tracker, a small image is extracted according to thetracked bounding box to be classified. By understanding the role of eachperson, the activity analysis is tuned to focus on the patient as theprimary “actor” in the scene and utilize the caregivers intosupplementary roles.

Patient Pose Classification and Context Detection:

Next, the pose of the patient is estimated, and so a pre-trained networkis fine-tuned to classify the depth images into one of the followingcategories: lying-down, sitting, or standing. Depth is prioritized overcolor as this is a geometric decision. To supplement the finalrepresentation, a real-time object detector is applied to localizeimportant objects that supplement the state of the patient, such as: bedupright, bed down, and chair. By combining bounding boxes identified aspeople with bounding boxes of objects, the present invention may betterascertain if a patient is, for example, “lying-down in a bed down” or“sitting in a chair”.

Motion Analysis:

Finally, in-place body motion is computed. For example, if a patient islying in-bed for a significant period of time, clinicians are interestedin how much exercise in-bed occurs. To achieve this, the mean magnitudeof motion is computed with a dense optical flow field within thebounding box of the tracked patient between successive frames in thesequence. This statistic indicates how much frame-to-frame, in-placemotion the patient is exhibiting.

Mobility Classification:

Table 1 describes a clinically-accepted 11-point mobility scale (ICUMobility Scale). This is collapsed into the Sensor Scale (left) of 4discrete categories. The motivation for this collapse was that when apatient walks, this is often performed outside the room where thesensors cannot see.

By aggregating the different sources of information described in thepreceding steps, attribute feature F_(t) is constructed with: 1. Was apatient detected in the image? (0 for no; 1 for yes); 2. What was thepatient's pose? (0 for sitting; 1 for standing; 2 for lying-down; 3 forno patient found); 3. Was a chair found? (0 for no; 1 for yes); 4. Wasthe patient in a bed? (0 for no; 1 for yes); 5. Was the patient in achair? (0 for no; 1 for yes); 6. Average patient motion value; 7. Numberof caregivers present in the scene.

These attributes were chosen because their combination describes the“state” of the activity. Given a video segment of length T, allattributes F=[F₁, F₂, . . . , F_(T)] are extracted and the meanF_(μ)=Σ_(t=1) ^(T)=F_(t)/T is used to represent the overall videosegment (the mean is used to account for spurious errors that mayoccur). A Support Vector Machine (SVM) is trained to automatically mapeach Fμ to the corresponding Sensor Scale mobility level from Table 1.

For healthcare provider assessment of patient mobility, the datagathered and analyzed can be transformed and presented in a number ofways. For example, visual representations of the room from the RGB-D canbe overlayed with heat maps of patient activity. One such representationof the data is a volumetric representation tracking people throughoutthe patient room, as illustrated in FIG. 3. FIG. 3 illustrates athree-dimensional view of a patient room with a heatmap of movementwithin the room, according to an embodiment of the present invention. Inanother representation of the data, an overhead view is provided to showmovement throughout the patient room. FIG. 4 illustrates a top down viewof a patient room with a heatmap of movement within the room, accordingto an embodiment of the present invention. Other transformations andrepresentations of the data are also included within the scope of thepresent invention. Examples include heatmaps of patient movement withinthe room or on a visual representation of the patient's body to showmovement of specific body parts/areas of the patient, volumetricrepresentations can also include layers of patient movement over atimeline of hours or days. Another example includes an image of the roomlabeled with patient activities that occurred in different areas of theroom. These examples are not meant to be considered limiting, and anyvisual representation of the data known to or conceivable to one ofskill in the art is included.

EXAMPLES

Several exemplary implementations of the present invention are included,in order to further illustrate the invention. These examples areincluded merely as illustrations of the present invention, and are notto be considered limiting. While implementation is contemplated in theICU, this is not to be considered limiting. The invention can beimplemented in any space where mobility tracking is needed. In oneexemplary implementation of the present invention, data were collectedwith three RGB-D sensors mounted on the walls of a single privatepatient room in the ICU to permit views of the entire room withoutobstructing clinical activities, as illustrated in FIG. 5. FIG. 5illustrates a sensor system, according to an embodiment of the presentinvention in an ICU room and example color (converted to grayscale fordemonstration) and depth images captured by the sensors. The grayscaleimage on the left provides texture information for human/objectdetection. Faces are obscured and the image is blurred for identityprotection. The depth image on the right shows the distance from thecamera to the human/object with darker gray pixels indicating areascloser to the camera, lighter gray pixels indicating areas farther awayand black pixels indicating that the depth camera cannot capture thedistance values around those regions. The depth image providescomplementary information for better human/object detection.

The sensors were activated and continuously captured color and depthimage data from the time of patient consent until ICU discharge. Examplecolor (converted to grayscale for demonstration) and depth imagesobtained from the sensor are shown in FIG. 5. Each sensor was connectedto a dedicated encrypted computer containing a storage drive. The datawere de-identified at the local storage drive, and then transferred,using a secure encrypted protocol, to the server for a second level ofobfuscation, storage and analysis.

The present invention automatically analyzes the sensor color and depthimage data to measure patient mobility and assign the highest level ofmobility within a time period. To characterize the highest level ofmobility numerically, a validated 11-point mobility scale was collapsedinto the 4 mutually exclusive mobility categories of Table 2. Walkingcategories were collapsed because the sensor only measures movementwithin the ICU room. As such, if a patient walks, then this is oftenperformed outside of the room. The remaining categories were collapsedbecause the data set, though it included a significant number of hoursof sensed data, did not include a sufficient number of mobility eventsspecific to each discrete mobility categories in the 11-point scale.

362 hours of sensor color and depth image data were recorded and curatedinto 109 segments, each containing 1000 images, from 8 patients. Thesesegments were specifically sampled to ensure representation of each ofthe mobility levels. Of the 109 segments, the present invention wasdeveloped using 26 of these from 3 ICU patients (“development data”) andvalidated on 83 of the remainder segments obtained from 5 different ICUpatients (“validation data”).

The algorithmic procedures performed for mobility measurement, withrespect to the present example are shown in FIG. 6B and described below.FIG. 6A illustrates images from a patient room according to anembodiment of the present invention. The images of FIG. 6A includeoverlaid bounding boxes to indicate the positions of people in thepatient room as detected by the sensor. The flow chart of FIG. 6B showsthe stages of the algorithm of the present invention.

The algorithm employs the following five steps: 1) analyze individualimages to locate the regions containing every person in the scene(Person Localization), 2) for each person region, assign an identity todistinguish ‘patient’ vs. ‘not patient’ (Patient Identification), 3)determine the pose of the patient, with the help of contextualinformation (Patient Pose Classification and Context Detection), 4)measure the degree of motion of the patient (Motion Analysis), and 5)infer the highest mobility level of the patient using the combination ofpose and motion characteristics (Mobility Classification).

The present invention was developed using “bounding boxes” of people andobjects in the development data (FIG. 6A). A bounding box is defined asa region of an image containing a person or object. For the developmentdata, a researcher annotated for who the people were in each image(patient vs. not-patient) as well as where objects were located (bed orchair). Using these annotations, the present invention was trained toautomate each of the 5 steps described below.

Given a segment of images, each image was analyzed independently and inorder. For each image, the present invention identified all regionscontaining persons using three steps. First, a collection ofperson-detection algorithms was used to identify candidate locations forpersons in each image. Second, these outputs were combined to obtain thehigh likelihood locations. Finally, false detections are further removedby imposing consistency checks for locations found in consecutiveimages. The result of this step was bounding boxes around persons in theimage (FIG. 6A).

Next, for all persons identified in an image, the present inventiondetermined whether they are a ‘patient’ or ‘not patient’ (e.g. caregiveror family member). This was done via a Convolutional Neural Network(CNN) algorithm. A CNN is a machine learning algorithm that can betrained to classify inputs into a specific class of outputs (e.g., imageregions into person vs. not), and then given a bounded region of aperson, the algorithm can automatically determine whether or not theyare a patient (FIG. 6B). The CNN achieves this automatically throughlearning characteristics of people's appearance based on color andgeometry.

Once both location and identity of the people in each image wereestablished, next their pose was characterized for the purpose ofmobility labeling. Poses were classified into 4 discrete categories: (1)lying in bed, (2) sitting in bed, (3) sitting in chair, and (4)standing. A pose detector was trained using the CNN algorithm thatautomatically learned the pose of a person. Using annotations from thedevelopment data, the CNN was trained to determine if a bounded regionof an image contained a person who was “lying-down”, “sitting”, or“standing”. Besides pose, an object detection algorithm was used toautomatically locate the bounded regions of objects in the images thatcorrespond to “beds” and “chairs” (also called “object detections”).These were then combined to the get the patient's overall pose (FIG. 6Aand FIG. 6B).

After classifying the pose and context of a person identified aspatient, information about their movement was extracted by analyzingconsecutive images to measure motion. Specifically, for a boundingregion containing a person in a given image, the subsequent images inthe segment were analyzed within the same region and measured the meanand variance of the changes in image intensity per pixel. In addition,speed of movement was computed by measuring the total distance that theperson moved (as measured by the center of the bounding regions) dividedby the duration over which the movement was made (FIG. 6A and FIG. 6B).

In this final step, the information related to pose, context, and motioncomputed in the steps above was aggregated into a series of numbers(often termed “feature”) to determine the final mobility level accordingto the scale in Table 2. The feature contained the following values: 1)was a patient detected in the image?; 2) what was the patient's pose?;3) was a chair found?; 4) was the patient in a bed?; 5) was the patientin a chair?; 6) what was the average patient motion value?; and 7) howmany caregivers were present in the room? These features were used totrain a Support Vector Machine (SVM) classifier to automatically mapeach feature to the corresponding mobility level from Table 2.

TABLE 2 N = 3 Characteristics (development) N = 5 (validation) Age (y),Median (IQR) 67 (60-71) 67 (52-77) Male  1 (33%)  2 (40%) ICU length ofstay (d), Median (IQR)  5 (1-5)  3 (2-5) Type of Surgery Endocrine(Pancreatic) 1 1 Gastrointestinal 1 2 Gynecologic 1 Thoracic 1Orthopedics 1 APACHE II score, Median (IQR) 13 (12-28) 16 (10-21)

The validation data consisted of 83 segments from 5 patients. Thissample size was considered to be sufficient with a 5.22%margin-of-error. For validation, two junior and one senior physicianindependently reviewed the same segments and were blinded to theevaluation method of present invention, reporting the highest level ofpatient mobility visualized during each segment, according to the sensorscale (Table 2). In the 27% of visualizations exhibiting disagreement,these were re-reviewed and the majority opinion was considered as thegold standard annotation.

The performance of the present invention was assessed using a weightedKappa statistic that measured disagreement between the mobility leveloutput of the present invention and the gold standard annotation. Astandard linear weighting scheme was applied which penalized accordingto the number of levels of disagreement (e.g., predicting “A” whenexpecting “B” yielded a 33% weight on the error whereas a prediction of“A” when expecting “C” yielded a 67% weight on the error, and 100%weight when expecting “D”). The percentage of segments on which thepresent invention agreed with the gold standard annotation wascalculated. The weighted percent observed agreement was computed by oneminus the linear weighting of different levels of disagreement. Acontingency table was created to report the inter-rater agreement foreach mobility level.

Patient demographics for the exemplary implementation are detailed inTable 3. The number of segments annotated as each mobility level, asidentified by the physicians (gold standard), is 21 (25%) for “nothingin bed”, 30 (36%) for “in-bed activity”, 27 (33%) for “out-of-bedactivity”, and 5 (6.0%) for “walking”. Table 4 reports gold standardversus the present invention agreement for each mobility level. In 72(87%) of the 83 segments there was perfect agreement between the goldstandard and the automated score of the present invention. Of the 11discrepancies, 7 were due to confusion between “nothing in bed” and“in-bed activity”. The weighted percent observed agreement was 96%, witha weighted Kappa of 0.86 (95% confidence interval: 0.72, 1.00).

TABLE 3 Physician Score A. Nothing B. In-bed C. Out-of-bed D. WalkingTotal Sensor A. Nothing 18 (22%) 4 (5%) 0 0 22 (27%) Score B. In-bed 3(4%) 25 (30%) 2 (2%) 0 30 (36%) C. Out-of-bed 0 1 (1%) 25 (30%) 1 (1%)27 (32%) D. Walking 0 0 0 4 (5%) 4 (5%) Total 21 (26%) 30 (36%) 27 (32%)5 (6%) 83 (100%)

TABLE 4 Patient Motion Status Pose Object Motion i. Lying in bed withoutmotion Lying Bed down No ii. Lying in bed with motion Lying Bed down yesiii. Sitting in bed Sitting Bed upright N/A iv. Sitting in chair SittingChair N/A v. Standing without motion Standing N/A No vi. Standing withmotion Standing N/A Yes but no moving vii. Walking Standing N/A Yesmoving

The main source of difference in sensor and clinician agreement lies indifferentiating “nothing in bed” from “in-bed activity”. The differencewas due, in large part, to segments where the patient motion was subtle.When applying the physician-evaluated discrete scale to the sensor, ifthe patient's total body speed signature exceeds a threshold, then thesensor labels this mobility level as “in-bed activity”. Below thisthreshold, any body activity is labeled as “nothing in bed”. Theclinician's activity threshold, which differentiates “nothing in bed”from “in-bed activity”, is subjective and different from that of thesensor, which is quantitative and thus more reproducible. Some of thesewere challenging segments; therefore, these speed judgment discrepanciesare not necessarily sensor errors. The sensor rarely exhibited genuineerrors due to incorrect assessment of pose or confusing patientidentities with provider identities, as illustrated in FIG. 7. FIG. 7illustrates graphical and image views of how the present invention canincorrectly measure a patient's mobility when the caregiver is close tothe patient. Currently, the in-bed motion is measured by image intensitychanges in a bounding region containing the patient. This kind ofmeasurement is inaccurate if the bounding region of the caregiver isoverlapped with the one of the patient.

Patient mobility data derived from the present invention couldauto-populate the health record such that providers no longer mustsubjectively document a highest level of mobility. The present inventioncould continuously monitor patient mobility in an ICU room and generateand report a single numeric value representing the patient's highestmobility level during an hour's time frame, as illustrated in FIG. 8, aswell as more detailed data about specific activity levels and durations.FIG. 8 illustrates a graphical view of how the present inventioncontinuously measures patient mobility. The mobility level for a segmentof several images is assigned after analyzing patient motioncharacteristics over a defined time period. A single numeric value,along with its trend over time is akin to a vital sign, and could beused to stimulate patient mobility quality improvement activities. Forexample, data from the present invention could be used to providereal-time feedback to providers, patients and their caregivers regardinga patient's mobility status for comparison to a pre-determined activitygoal, prompting care changes to ensure that patients are on track toachieving these.

As privacy may pose a concern, family members and providers werereassured by the fact that data was de-identified and encrypted both atthe local storage drive and server. They also often expressed comfortwith the general presence of sensors given their ubiquity in publicareas, expressing gratitude that the sensors were being used to ‘dogood’ and improve the quality of care delivery.

The analysis of the present invention is at the segment level and not atthe patient level, and as Table 3 demonstrates, the present inventionwas exposed to many different variations of human activities from lyingmotionless in a bed to small motions in bed, sitting up, and walking.Second, though patient and provider privacy was not an issue in theexemplary implementation, further studies are needed to establish thedegree of stakeholder comfort with sensing technologies such as thoseused in this study.

Sensor technology and deep machine learning techniques are used in otherindustries, but have only recently been explored in health care. Thepresent invention uses inexpensive technology and novel machine learningand computer vision-based algorithms to capture patient mobility in theICU. The results suggest that new deep learning techniques in machinelearning hold promise to automate activity recognition and sceneunderstanding. Other potential applications include delirium assessment(e.g., delirium motoric subtype), patient-provider interaction (e.g.,how physicians interact with patients in their room), and evaluation ofpatient turning in bed (e.g., as part of prevention efforts for pressureulcers). Adapting these techniques for clinical intervention monitoringoffers the potential for improving care measurement and delivery. Thenext steps include algorithmic refinements, applying the presentinvention to measure and provide performance feedback to providers andextending the repertoire of clinical tasks.

An accurate method for automating measurement of patient mobility hasbeen developed and evaluated in the ICU using RGB-D sensors, machinelearning and computer vision technologies. The present inventionaddresses a need for effective, inexpensive, continuous evaluation ofpatient mobility to assist with optimizing patient mobility in the ICU.

The steps and analysis of the present invention can be carried out usinga computer, non-transitory computer readable medium, or alternately acomputing device or non-transitory computer readable medium incorporatedinto the imaging device. Indeed, any suitable method of calculationknown to or conceivable by one of skill in the art could be used. Itshould also be noted that while specific equations are detailed herein,variations on these equations can also be derived, and this applicationincludes any such equation known to or conceivable by one of skill inthe art. A non-transitory computer readable medium is understood to meanany article of manufacture that can be read by a computer. Suchnon-transitory computer readable media includes, but is not limited to,magnetic media, such as a floppy disk, flexible disk, hard disk,reel-to-reel tape, cartridge tape, cassette tape or cards, optical mediasuch as CD-ROM, writable compact disc, magneto-optical media in disc,tape or card form, and paper media, such as punched cards and papertape. The computing device can be a special computer designedspecifically for this purpose. The computing device can be unique to thepresent invention and designed specifically to carry out the method ofthe present invention. The computing device can also take the form of anoperating console computer for the imaging device. The operating consoleis a non-generic computer specifically designed by the imaging devicemanufacturer for bilateral (input output) communication with the device.It is not a standard business or personal computer that can be purchasedat a local store. Additionally this console computer carries outcommunications with the scanner through the execution of proprietarycustom built software that is designed and written by the scannermanufacturer for the computer hardware to specifically operate thescanner hardware.

The many features and advantages of the invention are apparent from thedetailed specification, and thus, it is intended by the appended claimsto cover all such features and advantages of the invention, which fallwithin the true spirit and scope of the invention. Further, sincenumerous modifications and variations will readily occur to thoseskilled in the art, it is not desired to limit the invention to theexact construction and operation illustrated and described, andaccordingly, all suitable modifications and equivalents may be resortedto, falling within the scope of the invention.

What is claimed is:
 1. A method of measuring patient mobilitycomprising: analyzing individual images from an RGB-D sensor to locate aregion containing each person including a patient; assigning an identityfor each person in the region, to distinguish ‘patient’ vs. ‘notpatient’; determining a pose of the patient, with contextualinformation; measuring a degree of motion of the patient; and inferringa highest mobility level of the patient using the combination of poseand motion characteristics.
 2. The method of claim 1 further comprisinggenerating a volumetric representation of patient mobility.
 3. Themethod of claim 1 further comprising generating a heatmap of patientmobility.
 4. The method of claim 1 further comprising generating aheatmap of movement of regions of the patient's body.
 5. The method ofclaim 1 further comprising classifying the pose of the patient into 4discrete categories: (1) lying in bed, (2) sitting in bed, (3) sittingin chair, and (4) standing.
 6. The method of claim 1 further comprisingclassifying patient motion as “in-bed activity” if a total body speedsignature of the patient exceeds a threshold.
 7. The method of claim 1further comprising classifying patient motion as “nothing in bed” if atotal body speed signature of the patient is below a threshold.
 8. Themethod of claim 1 further comprising executing the method with anon-transitory computer readable medium.
 9. The method of claim 1further comprising defining a predetermined area for measuring patientmobility.
 10. The method of claim 9 further comprising defining thepredetermined area for measuring patient mobility as the patient room.11. A system for measuring patient mobility comprising: an RGB-D sensor;a non-transitory computer readable medium programmed for: analyzingindividual images from the RGB-D sensor to locate a region containingeach person, including the patient; assigning an identity for eachperson in the region, to distinguish ‘patient’ vs. ‘not patient’;determining a pose of the patient, with contextual information;measuring a degree of motion of the patient; and inferring the highestmobility level of the patient using the combination of pose and motioncharacteristics.
 12. The system of claim 11 further comprisinggenerating a volumetric representation of patient movement.
 13. Thesystem of claim 11 further comprising generating a heatmap of patientmovement.
 14. The system of claim 11 further comprising generating aheatmap of movement of regions of the patient's body.
 15. The system ofclaim 11 further comprising classifying the pose of the patient into 4discrete categories: (1) lying in bed, (2) sitting in bed, (3) sittingin chair, and (4) standing.
 16. The system of claim 11 furthercomprising classifying patient motion as “in-bed activity” if a totalbody speed signature of the patient exceeds a threshold.
 17. The systemof claim 11 further comprising classifying patient motion as “nothing inbed” if a total body speed signature of the patient is below athreshold.
 18. The system of claim 11 further comprising assigning anumerical mobility value to the patient mobility.
 19. The system ofclaim 11 further comprising defining a predetermined area for measuringpatient mobility.
 20. The system of claim 19 further comprising definingthe predetermined area for measuring patient mobility as the patientroom.